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Why  Trust  Matters 
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■  Congressional  Mandate 

■  The  National  Defense  Authorization  Act  for  Fiscal  Year 
2001,  Public  Law  106-398,  Congress  mandated  in 
Section  220  that  “It  shall  be  a  goal  of  the  Armed 
Forces  to  achieve  the  fielding  of  unmanned,  remotely 
controlled  technology  such  that...  by  2015,  one-third 
of  the  operational  ground  combat  vehicles  are 
unmanned.”1 
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Why  Trust  Matters 
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■  Secretary  of  Defense  Gates’  Comments 

■  On  21  April  2008,  Secretary  Gates  made  the  following  comment 
about  unmanned  systems 

“Unmanned  systems  cost  much  less  and  offer  greater  loiter  times 
than  their  manned  counterparts,  making  them  ideal  for  many  of 
today's  tasks.  Today,  we  now  have  more  than  5,000  UAVs,  a  25-fold 
increase  since  2001.  But  in  my  view,  we  can  do  and  we  should  do 
more  to  meet  the  needs  of  men  and  women  fighting  in  the  current 
conflicts  while  their  outcome  may  still  be  in  doubt.  My  concern  is  that 
our  services  are  still  not  moving  aggressively  in  wartime  to  provide 
resources  needed  now  on  the  battlefield.  I've  been  wrestling  for 
months  to  get  more  intelligence,  surveillance  and  reconnaissance 
assets  into  the  theater.  Because  people  were  stuck  in  old  ways  of 
doing  business,  it's  been  like  pulling  teeth.  ’2 
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Trust  Matters 
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Why  Trust  Matters 


■ 


■ 
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■  President’s  Budget  for  Unmanned  Systems3 


PORs 

FY09PB 

($M) 

Funding 

Source 

FY09 

FY10 

FY11 

FY12 

FY13 

TOTAL 

RDT&E* 

$1291.2 

$747.5 

$136.2 

SI  08.7 

$68.9 

$2,353 

UGV 

PROC* 

$33.4 

$42.3 

$53.5 

$59.5 

$21.1 

3210 

O&M* 

$2.9 

$3.9 

$3.0 

$12.8 

$10.1 

333 

RDT&E 

SI  347.0 

$1305.1 

$1076.4 

S894.0 

$719.5 

$5,342 

UAS 

PROC 

$1875.5 

$2006.1 

$1704.7 

$1734.3 

$1576.2 

$8,897 

O&M 

SI  54.3 

$251 .7 

S249.0 

$274.9 

$320.2 

$1,250 

RDT&E 

$57.3 

$73.8 

$63.2 

$70.1 

$76.9 

3341 

UMS 

PROC 

$56.7 

$78.4 

$95.9 

$91.6 

$103.7 

3426 

O&M 

$5.0 

$4.5 

$11.3 

$13.5 

$13.9 

348 

TOTAL 

$4,823 

34,513 

$3,393 

$3,260 

32,911 

$18,900 

*  RDT&E  =  Research,  Development,  Test,  and  Evaluation;  PROC  - 
Maintenance 

=  Procurement;  O&M  =  Operations  and 
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Why  Trust  Matters 
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Common  Performance  Envelope  Across  Domains3 
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Why  Trust  Matters 
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■  P.W.  Singer  of  the  Brookings  Institution  stated  the  following  in 
an  article  published  in  2009 

■  “So,  despite  what  one  article  called  ,&ll  the  lip  service  paid  to 
keeping  a  human  in  the  loop,  "the  cold,  hard,  metallic  reality  is 
that  autonomous  armed  robots  are  coming  to  war.  They  simply 
make  too  much  sense  to  the  people  that  matter. ”4 
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Why  Trust  Matters 
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■  From  recent  news,  the 
X-37B  Orbital  Test 
Vehicle  was  launched 
on  22  April  2010  and 
received  a  fair  amount 
of  media  attention 

■  Who  is  in  control? 

■  What  are  its 
capabilities? 

■  How  long  will  it 
remain  in  orbit  and 
what  is  it  doing? 


NASA 
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Why  Trust  Matters 
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■An  article  in  the  February  2010  issue  of 
Popular  Mechanics  asked  “Can  We  Trust 
Robots?”6 

■  The  primary  point  of  the  article  was  that  humans 
may  be  too  trusting  of  robots  because  they 
anthropomorphize  them 

■  We  humans  think  we  understand  a  robot's  “thought 
processes”  and  “motivations” 

■  This  could  become  worse  as  robots  become 
ubiquitous  and  as  we  begin  to  rely  upon  them  more 


29  April  2010 


Systems  &  Software  Technology  Conference  2010 


11 


Autonomy  Defined 
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■  The  American  Heritage  Dictionary  defines  autonomy 


as 


■  “Independence.  ” 7 

■  Wikipedia  defines  an  autonomous  agent  as 

■  “A  system  situated  in,  and  part  of,  an  environment,  which 
senses  that  environment,  and  acts  on  it,  over  time,  in 
pursuit  of  its  own  agenda.  This  agenda  evolves  from  drives 
(or  programmed  goals).  The  agent  acts  to  change  the 
environment  and  influences  what  it  senses  at  a  later  time.  ” 

■  “Non-biological  examples  include  intelligent  agents, 
autonomous  robots,  and  various  software  agents,  including 
artificial  life  agents,  and  many  computer  viruses.  ”  8 
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Autonomy  Defined 
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■  There  are  varying  degrees  of  autonomy 

■  None  -  System  is  completely  manually  controlled 

■  Partial  -  Some  functions  are  automated 

■  Sliding  -  The  amount  of  autonomy  is  selectable 

■  Full  -  The  system  operates  entirely  without  human  control 

■  With  any  autonomy  the  user  gives  up  some  level  of  control 

■  Wikipedia  has  a  good  definition  of  the  abilities  of  a  fully 
autonomous  robot 

■  “A  fully  autonomous  robot  has  the  ability  to 

•  Gain  information  about  the  environment 

•  Work  for  an  extended  period  without  human  intervention 

•  Move  either  all  or  part  of  itself  throughout  its  operating  environment 
without  human  assistance 

• Avoid  situations  that  are  harmful  to  people,  property  or  itself  unless 
those  are  part  of  its  design  specifications.  ” 9 


■ 
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Autonomy  Defined 
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Examples  of  Degrees  of  Autonomy 

Sliding  Autonomy 

Partial  Autonomy 

No  Autonomy  (ISS  assembly  ^ 
(Remote-control  robot) 


Full  Autonomy 
(ALCM) 


Boeing  Photo 


US  Air  Force  Photo 


NASA  Photo 


US  Air  Force  Photo 
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Trust  Defined 
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■  The  “classic”  definition  of  trust  comes  from  Diego 
Gambetta 

■  “trust  (or,  symmetrically,  distrust)  is  a  particular  level  of  the 
subjective  probability  with  which  an  agent  assesses  that 
another  agent  or  group  of  agents  will  perform  a  particular 
action,  both  before  he  can  monitor  such  action  (or 
independently  of  his  capacity  ever  to  be  able  to  monitor  it) 
and  in  a  context  in  which  it  affects  his  own  action.  When  we 
say  we  trust  someone  or  that  someone  is  trustworthy,  we 
implicitly  mean  that  the  probability  that  he  will  perform  an 
action  that  is  beneficial  or  at  least  not  detrimental  to  us  is 
high  enough  for  us  to  consider  engaging  in  some  form  of 
cooperation  with  him.”10 
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Research  Objective 
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■  Assuming  that  autonomous  agents  are  coming  to 
hazardous  environments,  like  disaster  areas  and 
combat  zones,  how  can  we  trust  them? 

■  This  research  project  is  attempting  to  identify  the 
factors  that  contribute  to  trust  in  autonomous  agents, 
in  order  to  develop  a  set  of  key  performance 
parameters  (KPPs)  for  trusted  agents 

■  These  KPPs  can  be  used  by  intelligent  agent 
developers  to  verify  the  trustworthiness  of  their  agents 
and  to  convince  users  of  their  trustworthiness 
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Research  Objective 
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■  Three  elements  to  the  research 

■  Identify  factors  of  trust  published  in  the  literature 

■  Simulate  human-agent  interactions 

■  Collect  data  on  human  interactions  with  autonomous 
agents 

■  The  ultimate  objective  is  to  develop  a  process  that 
can  be  validated  in  field  trials 
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Identifying  Trust  Factors 
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■  Cristiano  Castelfranchi  and  Rino  Falcone  argue  that 

Only  agents  endowed  with  goals  and  beliefs 
(cognitive  agents)  can  “trust”  another  agent11 

■  They  define  the  elements  of  trust  with  the  following  diagram 


GOAL  g 

Bi:  y  can  g,  has  the  power  of  g  (Evaluation) 

B2:  y  will-doa  for  g  (Expectation) 

B3:  g  will  be  true  (Tmst  that  g)  CORE  TRUST 


B4:  I  need  y  for  g  (Dependence) 

GOAL  of  not  doing/  not  exploit  alternatives' 
beting  on  y  (Reliance  and  bet) 

GOAT,  that  v  can  &  will  do _ RELIANCE^ 

Mental  ingredients  of  TRUST 
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Identifying  Trust  Factors 
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■  Sarvapali  Ramchurn,  et  al.,  identified  two  principle 
components  of  trust  in  an  agent12 

■  Confidence  -  Do  I  believe  the  agent  can  perform  the 
desired  task? 

■  Reputation  -  Has  this  agent  been  successful  in  the 
past  and  have  others  trusted  this  agent,  with  good 
results? 
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Identifying  Trust  Factors 
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■  Karen  Fullam  and  Suzanne  Barber  focused  on  the 
importance  of  reputation  when  dealing  with  agents 
(either  human  or  artificial)  in  the  development  of  the 
ART  Testbed13 
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Identifying  Trust  Factors 
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■  Just  in  these  three  research  papers,  the  following 
potential  trust  factors  were  identified 

■  Evaluation 

■  Expectation 

■  Trust 

■  Dependence 

■  Reliance 

■  Bet 

■  Confidence 

■  Reputation 

■  Which  factors  are  the  key  factors? 
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Simulating  Trust 
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■  Trust  of  autonomous  systems  is  an  active  area  of 
research 

■  Much  of  the  research  uses  simulations  to  explore 
issues  of  trust  and  to  compare  approaches 

■  The  simulations  tend  to  take  two  forms 

■  Simulations  designed  specifically  for  examining  issues 
of  trust 

■  Simulations  that  were  designed  for  other  areas  of 
research  that  have  been  extended  or  adapted  to  trust 
research 

■  The  following  are  some  examples  of  both  types  of 
simulation 
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Simulating  Trust 
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■Agent  Reputation  and  Trust  (ART)  Testbed14 

■  A  simulation  of  multiple  art  appraising  agents 
specifically  designed  for  agent  trust  research 


'Game  Monitor  Interlace:  Game  1 


* 


Agent  3 
S  76 


& 


6 


- ^ 

Opinion  Requests 
Opinions 

Reputation  Requests 

Reputations 

tjaSl 


SysiemData  Agent  1 


Bank  Balance 


Ti  roast-ip 

Transaction  Fees  Exchanged  by 
Timestep 


bB 


isiiir 


|  k  CSerA  Earnings 
|k  Opinion  Earwigs 
■  k  Reputation  Earnings 
k  Opinion  Costs 
v  Reputation  Costs 
!  k  Opmion  Senswg  Costs 


I  )  i  I  I  7  I  i 
Timstep 

Reputation  Weights 


■ 


29  April  2010 


Systems  &  Software  Technology  Conference  2010 


23 


Simulating  Trust 
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■  RoboCup  Rescue  Simulation15 

■  An  agent  simulation  system  that  has  been  used  for 
trust  research16 


Resource  Allocation 
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Simulating  Trust 
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■  Firefighting  Simulation 

■  A  simulation  of  human-robot  trust  relationships 
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Human-Robot  Trust  Game 

Prototype 
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■  A  prototype  of  a 
game  for  collecting 
user  interaction 
data  was  created  in 
NetLogo 

■  The  objective  of 
the  prototype  was 
to  evaluate 
playability  and 
design  elements 
for  data  collection 
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V  Human-Robot  Trust  Game 

Prototype 
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■  Last  year  a  paper-based  firefighting  game  was  play 
tested  at  the  RoboCup  International  Rescue  Robotics 
Workshop 

■  Based  on  feedback  from  the  playtesting,  a  print 
and  play  game  is  being  developed  to  gain  wider 
feedback  on  the  game  mechanics 

■  A  print  and  play  game  simulating  employment  of 
UASs  is  also  in  preliminary  development,  with  a 
working  title  of  Battle  for  Marjeh 

■  Links  to  both  games  and  a  wiki  for  feedback  will 
be  posted  to  the  project  web  page  when  they  are 
ready  (see  contacts  slide  at  end  of  presentation) 
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Web-based  Game 
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■  To  collect  data  from  a  variety  of  users,  a  web-based 
game  will  be  developed,  based  on  feedback  from  the 
print  and  play  game,  providing  tools  for  collecting 
data  on  how  game  players  utilize  the  robots 


■ 
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Diplomacy  Agent 
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■As  a  parallel  research  effort,  an  autonomous 
Diplomacy  playing  agent  is  going  to  be  developed  to 
explore  trust  elements  in  an  environment  intended  to 
encourage  distrust 

m  Diplomacy  games  are  available  as  play-by-email  and 
via  Internet  play  with  a  number  of  variants 

■  There  is  also  a  Diplomacy  Al  Centre  in  the  UK  that  is 
open  to  all  researchers  to  test  Diplomacy  agents  in 
piay  against  each  other  in  a  test  environment  called 
the  Diplomacy  Al  Development  Environment 
(DAIDE)17 
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Diplomacy  Agent 
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•  Diplomacy  was  created  in 
1954  by  Allan  B.  Calhameras 
a  simulation  of  the  diplomacy 
between  the  major  powers 
prior  to  World  War  I  in  Europe 

■  The  game  typically  has  seven 
players 

■  In  each  turn,  players 
participate  in  a  negotiation 
phase,  then  all  movements 
are  resolved  simultaneously18 


Diplomacy  Map  by  Martin  Asal 
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V  Identifying 

Human-Agent  Trust  KPPs 
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■  The  final  objective  for  this  project  is  to  develop  Key 
Performance  Parameters  that  can  be  used  by 
autonomous  agent  designers 

■  Serve  as  design  guidelines  for  agents 
■  Provide  verification  parameters  for  testing 
■  Provide  validation  parameters  for  field  testing 
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Summary 
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■  Why  Trust  Matters 

■  Autonomy  Defined 

■  Trust  Defined 

■  Identifying  Elements  of  Trust 

■  Simulating  Trust 

■  Human-Robot  Trust  Games 

■  Web-based  Game 

■  Diplomacy  Agent 

■  Identifying  Human-Agent  Trust  KPPs 
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Project  Contact  Info 
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daniel.stormont@aggiemail.usu.edu 


Human-Robot  Trust  Project  webpage: 
https://sites.google.eom/a/aggiemail.usu.edu/ 

human-robot-trust-project/ 


■ 
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Questions? 
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■  Al 

■  ALCM 

■  ART 

■  DAIDE 
■ DARPA 

■  ISS 

■  KPP 

■  NASA 

■  OSD 

■  SMXS 

■  UAS 

■  UAV 

■  UGV 

■  UMS 
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Artificial  Intelligence 
Air  Launched  Cruise  Missile 
Agent  Reputation  and  Trust 
Diplomacy  Al  Development  Environment 
Defense  Advanced  Research  Projects  Agency 
International  Space  Station 
Key  Performance  Parameter 
National  Aeronautics  and  Space  Administration 
Office  of  the  Secretary  of  Defense 
Software  Maintenance  Squadron 
Unmanned  Aerial  System 
Unmanned  Aerial  Vehicle 
Unmanned  Ground  Vehicle 
Unmanned  Maritime  System 
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